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Brief #4: Ensuring Fidelity of Assessment and Data Entry Procedures 


This brief discusses the importance of fidelity within 
assessment administration, assessment evaluation, 

and data entry procedures. It also includes methods 
to maintain fidelity within these practices. 


Schools can also learn more about the role of screening 
in NCRTI’s Essential Components document (National 


Center on Response to Intervention, 2010), http://www. 
rti4success.org/pdf/rtiessentialcomponents_042710. 
pdf, and from Module 1 (Screening) of the NCRTI 
Implementer Series http://Awww.rti4success.org/ 
resourcetype/rti-implementer-series-modules. 


All of the Screening Briefs in this series are available for 
download from NCRTI’s website, www.rti4success.org. 


Fidelity of Assessment 
and Data Entry Practices 


The term fidelity is often used in educational contexts 

to describe the degree to which a practitioner follows the 
prescribed directions when delivering an instructional 
program (Sanetti & Kratochwill, 2009; Dane & 
Schneider, 1998). Within an RTI framework, the term 
fidelity also extends to assessment and data entry 
practices (Mellard & Johnson, 2007). Fidelity within 
an assessment context is important; without 

it, it is difficult to determine whether results are 
an accurate representation of performance ora 
consequence of measurement error caused by 
inaccurate delivery of the assessment (NCRTI, 2011). 


Although all assessments have some measurement 
error, the amount of error often depends on the 
subjectivity of the assessment. Assessments requiring 
evaluation judgments are more susceptible to errors 
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than forced-choice assessments (e.g., where the 
student must choose a “right” or “wrong” answer). 
Minimizing the need for interpretation, and following 
standard procedures for delivering and scoring 
assessments, may help reduce this error (Sanetti & 
Kratochwill, 2009). 


Three Common Fidelity Challenges 


Problems with assessment fidelity typically come from 
three sources: assessment administration, assessment 
evaluation, and data entry (Taylor, 2009). Each 
category introduces a new type of measurement error 
into student scores, but it is possible to reduce the 
impact of this error through training. 


Category 1: Inconsistent Assessment Administration 


Unlike the other forms of assessment fidelity problems, 
administration errors tend to be systematic and can 
affect a large group of students. Administration errors 
occur when testers make mistakes while giving the 
assessment—for example, the tester gives incorrect 
instructions, provides inappropriate assistance, or alters 
assessment protocols (Taylor, 2009). A small protocol 
change—such as reducing time allocations or altering 
access to certain materials—may unnecessarily put 
some students at a disadvantage by making their scores 
appear inaccurately low. Conversely, providing 
additional assistance that violates administration 
instructions may inflate students’ scores, which can 
create the impression that at-risk students are not 

at risk. 


Administration errors often arise when individuals 
administering the assessment are unfamiliar with the 
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assessment or the assessment’s specific objectives. 
Alternatively, the administrator may be very familiar 
with the assessment, and his or her overconfidence 
may result in careless administration of the 
assessment. Thus, it is important to review 
administration procedures regularly so that all staff 
administer assessments in a consistent manner. 


Administration fidelity is improved by familiarizing 
the test administrator(s) with administration details 
and assessment objectives. This can be achieved 
through practice assessment sessions with a coach, 
or in a group session during professional development, 
followed by practice sessions with a coach. Using a 
checklist of the administration details also provides 

a review for administrator(s) prior to conducting 

the testing. 


Assessment companies often provide test manuals 
that contain assistive scripts, detailed directions, and 
frequently asked questions to help test administrators 
maintain continuity across administrations. These 
manuals often include information about permissible 
accommodations. In cases where accommodations are 
required, school officials should determine and plan 
for appropriate accommodations well ahead of the 
actual test administration. Some students may 
have accommodations as part of their Individualized 
Education Program (IEP) or 504 plan. For more 
information about testing accommodations, 
contact the test vendor and your state department 

of special education. 


Category 2: Errors in Scoring Assessments 


Scoring errors occur in situations where the evaluator 
incorrectly scores a student’s response, incorrectly 
interprets a student’s response, or is more lenient in 
scoring certain responses. It is important to follow the 
test publishers’ scoring rules to ensure consistency and 
minimize errors across evaluators. Evaluators can 
introduce bias while grading assessments—for 
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example, they may unconsciously project correct 
responses into student answers where none existed 
(Moskal & Leyden, 2000). Other errors may occur 
when two or more testers must evaluate an open- 
ended response. Here, even if steps have been taken 
to remove opportunities for interpretation, differences 
between testers may occur due to varying interpretations 
of the response. Even in cases where agreement has 
been established, scoring differences are likely to 
emerge as time passes (Moskal & Leyden, 2000). 


There are a number of steps that can be taken 
to reduce errors that may occur due to evaluator 
differences : 


© Code student names so that identification is 
difficult to prevent conscious or unconscious 
evaluation bias among testers. 


®  Inscenarios that require two or more evaluators, 
they should establish and follow written protocols 
prior to evaluating the assessments. 


e Assessment protocols should include rubrics for 
responses that require a degree of interpretation, 
and evaluators should be trained in the use of 
these rubrics with clear directions and assessment 
opportunities. 


e Once trained, evaluators should demonstrate high 
interrater reliability, which is the degree to which 
two raters, scoring identical student responses, 
agree with one another. 


e Evaluators should also periodically check to make 
sure they maintain high interrater agreement 
when scoring assessments. 


Category 3: Data Entry Errors 


Data entry errors are a third common source of 
assessment errors. Many assessment scores are now 
entered into databases via computer grading systems. 
Computer-based systems reduce many kinds of errors 
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and are generally the preferred method of data entry, 
although certain scenarios still require manual data 
entry. With manual data entry, scores may be 
incorrectly entered because of simple mistakes, but 
errors may also arise from disagreements in data entry 
procedures. A disagreement in data rounding 
procedures, for example, may lead to some students 
being categorized as at risk and others as making 
satisfactory progress. 


Data entry errors can be reduced through periodic 
data checks. One method of verification involves data 
being checked throughout the entry process. Once the 
data have been completely entered, staff should plan 
to randomly select a portion of the scores (e.g., 10-20 
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percent) for verification. Should data entry errors 
surface during the verification process, a deeper 
analysis is required to ensure an error-free database. 


Efforts to increase assessment fidelity of administration, 
scoring, and data entry will increase stakeholders’ 
confidence in the assessment system and reduce 
preventable classification errors. Establishing a system 
that minimizes error requires sufficient and ongoing 
training of staff members. Schools should work with 
assessment companies to ensure that assessment 
systems are designed and implemented properly at all 
levels, and that procedures are checked on an ongoing 
basis, to prevent drift and ensure implementation 
fidelity over time. 
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About the National Center on Response to Intervention 


Through funding from the U.S. Department of Education’s Office of 
Special Education Programs, the American Institutes for Research and 
researchers from Vanderbilt University and the University of Kansas 
have established the National Center on Response to Intervention. 
The Center provides technical assistance to states and districts and 
builds the capacity of states to assist districts in implementing 
proven response to intervention frameworks. 
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